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OVERVIEW 


©» olleges throughout the United States are evaluating the effectiveness of the strategies used 
_ to decide whether to place students into college-level or developmental education courses. 


© Developmental, or remedial, courses are designed to develop the reading, writing, or math 
skills of students deemed underprepared for college-level courses, a determination usually made 
through standardized placement tests. However, increasing numbers of colleges are using multiple 
measures to place students, including additional types of placement tests, high school transcripts, 
and evaluations of student motivation. 


There is no single, correct way to design and implement multiple measures assessment (MMA) to 
improve course placements. Colleges must decide what measures to include, and how to combine 
them. The current study was developed to add to our understanding about the implementation, cost, 
and efficacy of an MMA system using locally determined rules. As part of a randomized controlled 
trial, the study team evaluated MMA programs and interviewed and observed staff at five colleges 
in Minnesota and Wisconsin; it also wrote a short case study about one Wisconsin college. 


FINDINGS 


The five colleges in the random assignment study targeted all students taking placement tests in 
the months before the fall 2018 semester. In the four colleges included in the current analysis, 5,282 
students participated in the study; of these, 3,677 were tested for English, and 4,487 were tested for 
math. The findings suggest that while implementation (especially automation) was not easy, it was 
possible; and using the new MMA systems became much easier once they were established. 


Regarding the quantitative findings, in the first semester: 


¢ As intended, colleges used MMA to place program group students in their courses, with few 
exceptions. As a result, more program group students than control group students were referred 
to college-level gatekeeper courses, by 15 to 17 percentage points. 


¢ Program group students in the full sample also enrolled in more college-level gatekeeper courses 
than control group students (4.7 percentage points more in English; 3.9 percentage points more 
in math). 


¢ Students in the “bump up” zone — those eligible for college-level placement based only on MMA 
results, not a single standardized placement test — who placed into college-level English because 
they were in the program group were 28 percentage points more likely to have completed the 
gatekeeper English course by the end of their first college semester than their control group 
counterparts. 


¢ Students in the “bump up” zone who placed into college-level math were 12 percentage points 
more likely to have completed the gatekeeper math course by the end of their first college semester 
than their control group counterparts. 


The next and final report will present an analysis of transcript outcomes from three semesters of 
follow-up and will add two more cohorts to the research sample. 
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PREFACE 


» ~) tudents developing the academic and technical skills required in the current labor market 


» often rely on community colleges for their education. However, many of these students, some 
“eet of whom have been out of school for years, are academically underprepared. Students who 
place below college-level in English or math are typically placed in developmental courses that offer 
no college credit. More than half the community college students who place into developmental 
education do not graduate from a college-level program. Yet recent research suggests that many 
of these students may already be able to complete courses at the college level. Educators want to 
know whether relying on a single traditional placement test is making it harder for these students 
to succeed academically. 


To evaluate the predictive validity of single placement tests compared with “multiple measures” — 
the use of high school grade point averages, psychological assessments, or other appropriate crite- 
ria — MDRC teamed up with the Community College Research Center (CCRC), which carried out 
research in this area. MDRC and CCRC visited the Great Lakes region from 2015 to 2016 to better 
understand colleges’ interest in using multiple measures for course placements. The study team in 
2016 then launched the first phase of the Multiple Measures Assessment Project at 10 Minnesota 
and Wisconsin community colleges. 


An earlier MDRC publication, Toward Better College Course Placement: A Guide to Launching a 
Multiple Measures Assessment System, presents critical information, questions, and lessons gleaned 
from those efforts, with an emphasis on gauging institutional readiness, the importance of involv- 
ing the faculty in placement criteria decisions, integrating new measures into school systems, and 
refining conversations between advisers and students about placement results. 


The current phase of the project consists of a large randomized controlled trial of multiple measures 
assessments in 5 of the 10 pilot colleges in Minnesota and Wisconsin. In addition, MDRC and CCRC 
researchers, under the federally funded Center for the Analysis of Postsecondary Readiness, are 
evaluating multiple measures for placement at seven colleges in the State University of New York 
(SUNY) system. Early findings from the SUNY system came out in September 2018. The combined 
findings from these projects will provide causal evidence of the effects of using multiple measures 
placements on students’ completion of college courses. 


Virginia Knox 
President, MDRC 


Expanding Access to College-Level Courses ix 


ACKNOWLEDGMENTS 


F 


™)" he authors are thankful to the many administrators and faculty and staff members who helped 
| implement and evaluate new ways of placing students at the six participating colleges: Anoka 
_ Ramsey Community College, Century College, Madison College, Minneapolis Community and 
Technical College, Normandale Community College, and Northeast Wisconsin Technical College. 
We would also like to thank the Minnesota State and Wisconsin Technical College Systems for their 
cooperation and participation in this project. Thanks to Amy Kerwin and Sue Cui at Ascendium 
Education Group for their ideas and insight throughout the life of this project, and to Ascendium 
Education Group for its generous financial support of this project. 


We would like to thank current and recent members of the Multiple Measures Assessment team from 
MDRC and the Community College Research Center, including Rashida Welbeck, Alyssa Ratledge, 
Dorota Biedzio, Stanley Dai, and Kevin Thaddeus Brown, Jr. Thanks also to our senior advisers 
and reviewers — Thomas Brock, Michael Weiss, and Leigh Parise — for their careful reading and 
thoughtful feedback during the review process. We thank Will Swarts for editing this report and 
Carolyn Thomas for preparing it for publication. 


Expanding Access to College-Level Courses xi 


EXECUTIVE SUMMARY 


© olleges throughout the United States are evaluating the effectiveness of the strategies they use 
_____ to decide whether to place students into college-level or developmental education courses. 


Sd Developmental, or remedial, courses are designed to develop the reading, writing, or math 
skills of students deemed underprepared for college-level courses, a determination usually made 
through the use of a single placement test in each subject.’ Almost all colleges have used single 
placement tests to determine student course levels, but that is changing with the increased use of 
multiple measures — which may include additional types of placement tests, high school transcripts, 
evaluations of student motivation, and discussions with advisers — to assess and place students.? 
Research has generated a growing body of evidence demonstrating that single placement tests are 
highly inaccurate and that correct, academically appropriate placements are more likely when other 
measures, especially the high school grade point average (GPA), are taken into account.? 


But how much does this matter? It turns out that accurate placement can meaningfully influence 
students’ experiences and outcomes. Millions of students each year, about 55 percent of those en- 
tering community colleges, are placed into developmental education in math and English upon 
enrollment.’ These courses are intended to ensure that students acquire the necessary literacy and 
numeracy skills required for success with college-level courses. However, placing students into 
these courses delays their entry into credit-bearing coursework and earning a college credential.° 
Further, students who begin their studies in developmental education are less likely to graduate.® 
Thus, students should only take the developmental courses truly necessary to succeed in college 
coursework. Several studies suggest that existing referral systems based on single tests result in con- 
siderable underplacement in developmental courses, as well as some overplacement.’ Results show 
that underplaced students in developmental courses could have succeeded in credit-bearing college 
courses, and overplaced students wound up in courses they were highly unlikely to pass. One study 
found high rates of “severe” underplacement — 18 percent in developmental math and 25 percent in 
developmental English. These students were likely to have passed a college-level course with a B or 
better.® The study established that misplacement rates of all kinds could be reduced by employing 
multiple measures to determine the right course level for each student. 


1. Elizabeth Ganga, Amy Mazzariello, and Nikki Edgecombe. Developmental Education: An Introduction for 
Policymakers (New York: Community College Research Center, Columbia University, 2018). 


2. Elizabeth Zachry Rutschow and Alexander K. Mayer, Early Findings from a National Survey of Developmental 
Education Practices (New York: MDRC, 2018). 


3. Clive Belfield and Peter M. Crosta, “Predicting Success in College: The Importance of Placement Tests and High 
School Transcripts,” CCRC Working Paper No. 42 (New York: Community College Research Center, Columbia 
University, 2012); Judith Scott-Clayton, Do High-Stakes Placement Exams Predict College Success? CCRC Working 
Paper No. 41 (New York: Community College Research Center, Columbia University, 2012). 


4. Paul Attewell, David Lavin, Thurston Domina, and Tania Levey, “New Evidence on College Remediation,” The Journal 
of Higher Education 77, no. 5 (2006). 


5. Thomas Bailey, Shanna Smith Jaggars, and Davis Jenkins, Redesigning America's Community Colleges: A Clearer 
Path to Student Success (New York: Community College Research Center, Columbia University, 2015); Shanna 
Smith Jaggars and Georgia West Stacey, What We Know About Developmental Education Outcomes (New York: 
Community College Research Center, Columbia University, 2014). 


6. Jaggars and Stacey (2014). 
7. Belfield and Crosta (2012); Scott-Clayton (2012). 
8. Scott-Clayton (2012). 
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There is no single, correct way to design and implement multiple measures assessment (MMA) to 
improve course placements. Colleges must decide what measures to include, which means factoring 
in the difficulty of obtaining certain kinds of information about students, as well as how to com- 
bine the measures selected. The high school GPA is the most common measure used, along with 
placement test scores.? Other standardized test results, such as SAT and ACT test scores, and other, 
noncognitive assessments may also be considered.'° The relative importance of this information, 
and how it is evaluated to assess academic potential, must then be considered. Options range from 
a simple waiver system in which one or more criteria are used to allow students to forgo placement 
tests to using more complex methods, including using predictive models to place students based 
on their likelihood of success in the first college-level courses in English and math, also known as 
“gatekeeper” courses."' 


Limited prior research has examined the extent to which placement systems using multiple measures 
result in better college outcomes. The Community College Research Center (CCRC) and MDRC 
are conducting a random assignment evaluation of a predictive analytics assessment and placement 
system at seven State University of New York (SUNY) community colleges. Early findings indicate 
that the use of MMA can improve student outcomes in college.'? Other research by the RP Group 
in California,'? by the North Carolina Community College System, and by Ivy Tech Community 
College in Indiana further suggests that MMA is a promising approach. 


The current study was conducted in two midwestern states — Minnesota and Wisconsin — and 
sought to add to the knowledge base about the implementation, cost, and efficacy of an MMA system 
that uses a set of locally determined decision rules. The study asks these questions: 

1. What processes do colleges use to set up and implement an MMA system? 


2. What is the design of the MMA system at each college? 


3. What factors support or hinder high-quality implementation of the MMA system in each locale? 


9. Judith Scott-Clayton, Peter M. Crosta, and Clive R. Belfield, “Improving the Targeting of Treatment: Evidence from 
College Remediation,” NBER Working Paper 18457 (Cambridge, MA: National Bureau of Economic Research, 2012). 


10. Noncognitive assessments measure student qualities, characteristics, and attitudes, apart from content knowledge 
that may influence success in educational endeavors. Since these assessments require cognition, some people 
prefer other terms such as nonacademic, soft skill, or 21st century skills assessments. Examples include the College 
Board’s SuccessNavigator, ACT Engage, and the Grit Scale. 


11. Elisabeth A. Barnett and Vikash Reddy, College Placement Strategies: Evolving Considerations and Practices (New 
York: Community College Research Center, Columbia University, 2017). 


12. Elisabeth A. Barnett, Peter Bergman, Elizabeth Kopko, Vikash Reddy, Clive Belfield, and Susha Roy, Multiple 
Measures Placement Using Data Analytics: An Implementation and Early Impacts Report (New York: Community 
College Research Center, Columbia University, 2018). 


13. Mina Dadgar, Linda Collins, and Kathleen Schaefer, Placed for Success: How California Community Colleges Can 
Improve Accuracy of Placement in English and Math Courses, Reduce Remediation Rates, and Improve Student 
Success (San Rafael, CA: The RP Group Career Ladders Project, 2015). 
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4. How does using multiple measures to “bump up” student placements affect the rate of success- 


ful outcomes at these colleges? 


Implementation Findings 


For colleges considering scaling MMA to large numbers of students, the results drawn from the 
experiences of the colleges in this study offer some useful lessons. 


¢ Clear explanations of MMA systems help college stakeholders to understand and support the 
use of MMA. Colleges must have consistent messaging focused on how MMA could improve the 
school’s placement accuracy and student outcomes. This can help to garner support among faculty 
and the full range of staff involved in implementation. 


¢ There is a trade-off between more automated placement systems and more personalized pro- 
cesses found in systems that depend on interaction with advisers. Colleges in the study were 
moving toward greater automation, something that was encouraged in this project; however, they 
also were thinking about how to preserve opportunities for meaningful interactions between 
students and advisers. 


¢ The amount of staff time required to set up an MMA system is substantial but shrinks as the 
program is adopted. It may even result in time savings for staff once greater automation is used 
in placement decisions. 


¢ Timely access to high school GPA information remains a primary challenge in creating accurate 
MMA systems. In most cases, MMA implementation depends on students bringing transcripts 
to the college at the time of admission; however, this may not be the norm at some colleges. Some 
colleges are obtaining transcript data directly from local high schools, facilitating access to student 
data. It may also make sense to use student self-reports, given increasing evidence that students 
report their GPAs accurately. 


¢ Administering more than one test during the placement process can add challenges. It is im- 
portant to weigh the added difficulty of using a noncognitive assessment against its added value 
to the placement process. More information about the contribution of noncognitive assessments 
to better student placement determinations will be available in the final report. 


Measures Used and Placement Approach 


All colleges in the study included the following measures in their MMA systems: placement test 
scores, high school GPA, noncognitive assessment results, and scores from the ACT and SAT. The 
specific measures and decision rules used at each college are displayed in Table ES.1. 


14. In the MMA systems set up in this project, students could only be placed higher than they would be using a single 
measure, usually the placement test. Thus, they can be “bumped up.” 
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COLLEGE NAME 
AND STATE 
Anoka-Ramsey 
Community 
College, Minnesota 


Century College, 
Minnesota 


Madison College, 
Wisconsin 


Minneapolis 
Community and 
Technical College, 
Minnesota 


Normandale 
Community College, 
Minnesota 


Northeast 
Wisconsin Technical 
College, Wisconsin 


TABLE ES.1 MMA Approaches at Colleges in the 
Multiple Measures Assessment Study — Phase II 


TYPE OF 
PLACEMENT 
SYSTEM 


Decision rule 


Decision rule 


Decision band 


Decision band 


Decision rule 


Decision band 


MMA APPROACH 
AND ORDER OF STEPS 


1. Exemptions (AP/ 


IB, ACT, SAT, MCA 
scores) 


. ACCUPLACER 


(exemption) 


. GPA or LASSI 
. Exemptions (AP/ 


IB, ACT, SAT, MCA 
scores) 


. ACCUPLACER 


(exemption) 


. GPA or LASSI 
. Exemption (ACT 


score) 


. ACCUPLACER 


(decision band) 


. GPA or Grit 
. Exemptions (ACT, 


IB, SAT MCA scores, 
college credit) 


. ACCUPLACER 


(decision band) 


. GPA or LASSI 
. Exemptions (AP, ACT, 


SAT, MCA scores, 
college credit) 


. LASSI 
. GPA or 


ACCUPLACER 
(exemption) 


. Exemption (GPA) 


. ACT + Grit or 


ACCUPLACER + 
Grit (decision band) 


NONCOGNITIVE ASSESSMENT 
LASSI (motivation): 50th percentile 


LASSI (motivation): 50th percentile 


Grit Scale: 4+ 


LASSI (motivation): 75th percentile 


LASSI (motivation): 75th percentile 


Grit Scale (perseverance): 
3=1 pt; 

4=2 pts.; 

5+ = 3 pts. 


COLLEGE-READY HIGH 


SCHOOL GPA LEVEL 


English/Math: 
> 3.0 GPA 


English/Math: 
> 3.0 GPA 


English/Math: 
> 2.6 GPA 


English: 
> 2.3 GPA 


Reading: 
> 2.4 GPA 


Math: 

> 3.0 GPA 
English/Reading: 
>2.5 GPA 


Math: 
> 2.7 GPA 


English/Math: 
> 2.6 GPA 


NOTE: DECISION RULES are a sequence of rules that compares each selected measure with a threshold in a predetermined order. If the 
threshold is met, a placement is generated; if not, another rule is applied. DECISION BANDS are decision rules that apply only to students who 
fall within a certain range on a specified indicator (such as high school grade point average or a placement test score), usually just below the 


cutoff. 
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Once the colleges selected their assessment measures, they had to decide how those measures would 
be combined. This was usually done by developing a set of decision rules in which each measure 
would be considered in a specific order to determine which classes students were eligible to take. 
The colleges in the study sought to automate this process as much as possible. The third column 
in Table ES.1 shows the sequence in which colleges considered these measures. Typically, colleges 
considered waivers first to identify students who would be exempt from consideration of other 
measures. Subsequently, the results of the ACCUPLACER placement test,’® the high school GPA, 
and the noncognitive assessment would be considered. In some cases, a system of “decision bands,” 
applicable to students within a particular score range, was used. In these cases, students who earned 
test scores within a certain range would be evaluated using other measures. 


Identifying, Recruiting, and Randomly Assigning Students 


The five colleges participating in the random assignment study targeted all students taking placement 
tests following their admission to the colleges in the fall 2018 semester. Across the four Minnesota 
colleges,'® 5,282 students participated in the study, testing in English, math, or both. Of these, 3,677 
were tested for English, and 4,487 were tested in math. Students enrolling in college in spring 2019 
and fall 2019 are also participating in the study; however, the current report only includes findings 
from the first cohort (fall 2018). 


Effects of Multiple Measures Assessment 


This section presents the estimated impacts of the program at the end of the first semester for the 
first cohort of study students. These analyses seek to ascertain whether the students offered college- 
level course placement because of MMA are taking steps toward completion of a college-level course 
in math or English. The analyses in this interim report do not gauge the effectiveness of the changes 
in the placement system on the primary outcomes of interest (course completion and credit accu- 
mulation after three semesters), but they do provide insights into whether the short-term outcomes 
indicate that students are on track for success in later semesters. 


Summary of Findings 
In the first semester: 


¢ As intended, colleges used MMA to place program group students in their courses, with few 
exceptions. As a result, more program group students than control group students were referred 
to college-level gatekeeper courses, by 15 to 17 percentage points. 


15. ACCUPLACER is an assessment exam developed by the College Board to assess student skills in reading, math, 
and writing. It is widely used by U.S. two- and four-year colleges. 


16. The fifth college in the randomized controlled trial, from Wisconsin, randomized a large number of students, but 
because of implementation bottlenecks associated with a lack of automation in its placement process, a very small 
number of students were given the opportunity to be placed using multiple measures in the first program semester. 
Changes were made to improve this for the fall 2019 cohort, which will be included in the final report’s analysis, but 
for now, the fifth college is not included in the analysis. 
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¢ Program group students in the full sample were more likely to enroll in college (take one or more 


classes at the college where they tested) than control group students (2.5 percentage points more). 


¢ Program group students in the full sample also enrolled in more college-level gatekeeper courses 


than control group students (4.7 percentage points more in English; 3.9 percentage points more 


in math). 


¢ Students in the “bump up” zone who placed into college-level English were 28 percentage points 


more likely to have completed the gatekeeper English course by the end of their first college se- 


mester than their control group counterparts (Table ES.2). 


¢ Students in the “bump up” zone who placed into college-level math were 12 percentage points 


more likely to have completed the gatekeeper math course by the end of their first college semester 


than their control group counterparts (Table ES.3). 


TABLE ES.2 First-Semester College Transcript Outcomes 
Among Students in the English "Bump Up" Zone, 
Multiple Measures Assessment Study — Phase II 


PROGRAM CONTROL 
OUTCOME (%) GROUP GROUP DIFFERENCE P-VALUE 
English 
Placed in gatekeeper course 100.0 0.0 100.0 *** 0.000 
Enrolled in gatekeeper course 54.8 9.8 45.0 *** 0.000 
Completed gatekeeper course (C or higher) 34.5 6.7 27.8 *** 0.000 
Failed gatekeeper course 12.7 1 By al 0.000 
Withdrew from gatekeeper course 3.2 1.7 1.6 0.223 
Placed in developmental course 0.0 100.0 -100.0 *** 0.000 
Enrolled in developmental course 5.3 36.8 -31.5 ** 0.000 
Completed developmental course (C or higher) 4.3 29.4 -25.1 *** 0.000 
Failed developmental course 0.3 2.8 62,5 38" 0.006 
Withdrew from developmental course 0.5 2.2 ey a 0.051 
Enrolled in any course 83.0 75.2 V1 0.018 
Sample size (total = 624) 363 261 


SOURCE: Transcript data provided by Anoka-Ramsey Community, Century, Madison Area Technical, Minneapolis 
Community and Technical, and Normandale colleges. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 
Distributions may not add to 100 percent because categories are not mutually exclusive. 
= 1 percent; ** = 5 percent; * = 10 percent. 


RE 


Statistical significance levels are indicated as: 


The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention 


with zero true effect. 
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TABLE ES.3 First-Semester College Transcript Outcomes 
Among Students in the Math "Bump Up" Zone, 
Multiple Measures Assessment Study — Phase II 


PROGRAM CONTROL 


OUTCOME (%) GROUP GROUP DIFFERENCE P-VALUE 
Math 
Placed in gatekeeper course 100.0 0.0 100.0 *** 0.000 
Enrolled in gatekeeper course 26.6 2.9 23.7 *** 0.000 
Completed gatekeeper course (C or higher) 13.1 1.6 11.5 ** 0.000 
Failed gatekeeper course 3.7 0.4 3.3 7 0.003 
Withdrew from gatekeeper course 6.6 0.8 5.8 *** 0.000 
Placed in developmental course 0.0 100.0 -100.0 *** 0.000 
Enrolled in developmental course 44 274 -23.3 *** 0.000 
Completed developmental course (C or higher) 2.5 20.5 -17.9 *** 0.000 
Failed developmental course 1.2 4.8 -3.6 *** 0.004 
Withdrew from developmental course 0.2 1.4 -1.2 * 0.075 
Enrolled in any course 86.1 82.8 3.3 0.228 
Sample size (total = 703) 358 345 


SOURCE: Transcript data provided by Anoka-Ramsey Community, Century, Madison Area Technical, Minneapolis 
Community and Technical, and Normandale colleges. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

Distributions may not add to 100 percent because categories are not mutually exclusive. 

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention 
with zero true effect. 


Effects on Educational Outcomes After the First Semester 


The next and final report will present an analysis of transcript outcomes from three semesters of 
follow-up and will add two more cohorts to the research sample. That follow-up will enable com- 
parisons between groups after students who placed into developmental courses have had a chance 
to complete them and enroll in college-level courses. That report should offer more robust evidence 
about the type of placement systems that help students make better progress through their first 
three semesters of college."” 


17. The findings presented in this report are preliminary (and not “confirmatory”). The prespecified confirmatory 
outcomes on which the effectiveness of the program will be judged will be measured after three semesters, 
including two additional cohorts, and will be presented in the final report in 2021. These outcomes will include 
completion of the first college-level course (student completes the course with a grade of C or higher) within three 
semesters, by subject, and cumulative college-level credit accumulation within three semesters. 
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Introduction and Background 


»-) olleges throughout the United States are evaluating the effectiveness of their strategies used 
_. to place students into college-level or developmental education courses. Developmental, or 
remedial, courses are designed to develop the reading, writing, or math skills of students 
deemed underprepared for college-level courses, a determination usually made through the use of 
standardized placement tests such as the ACCUPLACER.' For years, colleges have used single place- 
ment tests, but that is changing with the increased use of multiple measures — other test scores, high 
school transcripts, and evaluations of student motivation — to assess and place students.” Research 
has generated a growing body of evidence demonstrating that single placement tests are highly inac- 
curate and that correct, academically appropriate placements are more likely when other measures, 
especially high school grade point averages (GPAs), are taken into account.® 


But how much does this matter? It turns out that accurate placement can meaningfully influence 
students’ experiences and outcomes. Millions of students each year, about 55 percent of those entering 
community colleges, are placed into developmental education in math and/or English upon enroll- 
ment.* These courses are intended to ensure students acquire the necessary literacy and numeracy 
skills required for success with college-level courses. However, placing students into these courses 
delays their entry into credit-bearing coursework and earning a college credential.° Further, students 
who begin their studies in developmental education are less likely to graduate.® Thus, students should 
only take the developmental courses truly necessary to succeed in college coursework. 


Several studies suggest that existing placement systems based on single tests result in consider- 
able underplacement in developmental courses, as well as some overplacement.’ Results show that 
underplaced students in developmental courses could have succeeded in credit-bearing college 
courses, and overplaced students wound up in courses they were highly unlikely to pass. One study 
found high rates of “severe” underplacement — 18 percent in developmental math and 25 percent in 
developmental English. These students were likely to have passed a college-level course with a B or 


1. Ganga, Mazzariello, and Edgecombe (2018). 

Rutschow and Mayer (2018). 

Belfield and Crosta (2012); Scott-Clayton (2012). 

Attewell, Lavin, Domina, and Levy (2012). 

Bailey, Jaggars, and Jenkins (2015); Jaggars and Stacey (2014). 
Ganga, Mazzariello, and Edgecombe (2018). 

Belfield and Crosta (2012); Scott-Clayton (2012). 
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better.® The study established that misplacement rates of all kinds could be reduced by employing 
multiple measures to determine the right course level for each student. 


Multiple Measures Assessment and Placement 


Multiple measures assessment (MMA) involves using two or more measures in combination to refer 
students to the most appropriate courses. There is no single, correct way to design and implement 
MMA to improve course placements. Colleges must decide what measures to include, factoring in 
the difficulty of obtaining certain kinds of information about students. Most often, the high school 
GPA is considered along with placement test scores.° Other standardized test results such as the SAT 
and ACT test scores, and other measures such as noncognitive assessments may also be included."° 


The relative importance of this information, and how it is evaluated in assessing academic potential, 
must then be determined. Options range from a simple waiver system in which one or more criteria 
are used to waive student placement tests to more complex methods, including using predictive 
models to place students based on their likelihood of success in the first college-level courses in 
English and math, also known as “gatekeeper” courses."' 


Limited prior research has examined the extent to which placement systems using multiple measures 
result in better college outcomes. The Community College Research Center (CCRC) and MDRC 
are conducting a random assignment evaluation of a predictive analytics assessment and placement 
system at seven State University of New York (SUNY) community colleges. Early findings indicate 
that the use of MMA can improve student outcomes in college.'* Other research by the RP Group 
in California,'? by the North Carolina Community College System, and by Ivy Tech Community 
College in Indiana further suggests that MMA is a promising approach. 


Focus of the Current Research 


Colleges are increasingly interested in MMA implementation, especially in the face of growing evi- 
dence of the value of high school GPA in predicting success in college. However, it is difficult to know 
what system to use. The research conducted on MMA systems using predictive analytics shows that 
they can improve student outcomes,” but the approach is complicated and requires sophisticated 


8. Scott-Clayton (2012). 
9. Clayton, Crosta, and Belfield (2012). 


10. Noncognitive assessments measure student qualities, characteristics, and attitudes, apart from content knowledge 
that may influence success in educational endeavors. Since these assessments require cognition, some people 
prefer other terms such as nonacademic, soft skill, or 21st century skills assessments. Examples include the College 
Board’s SuccessNavigator, ACT Engage, and the Grit Scale. 


11. Barnett and Reddy (2107). 

12. Dadgar, Collins, and Schaefer (2015). 
13. Barnett et al. (2018). 

14. Barnett et al. (2018). 
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analyses of historical data. The current research was specifically designed to examine whether a 
simpler MMA system can also positively influence student outcomes. 


In addition, many colleges are considering the use of noncognitive assessments as part of the place- 
ment determination, with the understanding that college success is not determined by content 
knowledge alone. Noncognitive assessments can be valuable sources of information about students’ 
readiness for college and may be particularly useful in cases where high school transcript data are 
unavailable or for nontraditional students who have been out of the education system for an extended 
time. However, very little information is available about whether existing noncognitive assessments 
are useful in making placement decisions. The current research will improve our understanding of 
their value in creating effective MMA systems. 


About the Study 


The current study was designed to address these areas of interest and improve the knowledge base 
on the implementation, cost, and efficacy of an MMA system that uses locally determined rules. 
The questions driving this study are as follows: 


1. What processes do colleges use to set up and implement an MMA system? 
2. What is the design of the MMA system at each college? 
3. What factors support or hinder high-quality implementation of the MMA system in each locale? 


4. How does using multiple measures to “bump up”® 


ful outcomes at these colleges? 


student placements affect the rate of success- 


The findings shared in this report are derived from a research project undertaken by MDRC and 
CCRC to study the use of MMA in Minnesota and Wisconsin, with funding from the Ascendium 
Education Group. Included in this report are preliminary results from the second phase of a two- 
phase project. In Phase I of this project,"® MDRC supported 10 colleges in Minnesota and Wisconsin 
as they created and piloted MMA placement systems using decision rules that were developed based 
on prior research and local knowledge; they all incorporated noncognitive assessments. A guidebook 
describing lessons learned during Phase I was created to help other colleges develop similar systems.'” 


Six colleges are participating in the current project, Phase II. At five of them, the research team 
provided technical assistance to college staff to create MMA systems incorporating locally deter- 
mined decision rules, many of which were based on those developed in Phase I. These five colleges 
went through considerable efforts to build systems to automate the placement process as much as 
possible to ultimately permit them to be scaled to the full student population. They also are engaged 


15. In the MMA systems set up in this project, students could only be placed higher than they would be using a single 
measure, usually the placement test. Thus, we say that they have the opportunity to be “bumped up.” 


16. October 2016 to March 2018. 
17. Cullinan et al. (2018). 
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in a randomized controlled trial comparing students placed using the college’s existing procedures 
with students placed using MMA. 


Colleges in this project began enrolling students into the study in the fall of 2018 and will continue 
to do so through the fall of 2019, a total of three semesters. Except for students who opt out (a rare 
occurrence), qualifying students enrolling at each college are randomly assigned to be placed using 
the MMA system or their college’s traditional placement system, typically using the ACCUPLACER 
placement test alone. Student outcomes in the two groups will be compared one semester following 
placement (included in this report) and two to three semesters following placement (in a forthcom- 
ing final report). 


MDERC and CCRC are also conducting implementation and cost research at six colleges, the five 
involved in the random assignment study and one more that participated in Phase I and decided to 
fully scale its MMA system right away. 


About This Report 


In this report, we describe the development of MMA systems at the participating colleges and share 
early impact findings from the first semester of implementation. Chapter 1 introduces the project, 
and Chapter 2 describes the process used to create MMA systems at the colleges, their designs, and 
the reasons for some of the design decisions made. Chapter 3 discusses system procedures as well 
as conditions that support or hinder high-quality implementation and potentially affect the impact 
on student outcomes. Chapter 4 presents early impact findings and includes a brief case study of 
the sixth college participating in this study. Chapter 5 considers the implications of this study for 
practice and future research. 


Publication of the final report associated with this project is anticipated in 2021, focusing on longer- 


term impacts of this MMA approach as well as a cost-effectiveness analysis and further analysis of 
the predictive utility of noncognitive assessments. 
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Planning and Designing Multiple Measures 
Assessment Systems 


™ his chapter addresses two research questions: how colleges involved in this study planned their 

multiple measures assessment (MMA) systems, and their initial design choices. The research 

team conducted one-day site visits to the six colleges participating in the study during October 

and November 2018 and spoke to representatives from admissions; testing; advising; information 

technology (IT); and the English, reading, and math departments; as well as college leadership.' The 

findings are supplemented by data from visits to the same colleges in summer 2017, during Phase 

I of the project. Phase I gave colleges the opportunity to consider and design the logistical aspects 

of an MMA system to ensure it worked as intended. Phase II involved the implementation of each 

college’s MMA system at scale, along with an analysis of its impact on student outcomes at five of 
the six colleges. 


Phase I: Initial Development of an MMA System 


Each college established an initial MMA system design during Phase I of this project. Colleges worked 
through design decisions individually and collaboratively, while receiving technical assistance from 
MDRC and the Community College Research Center (CCRC). The participating colleges piloted 
their systems with limited numbers of students, and generally did not change students’ placements 
during the pilot. 


The colleges took the following steps in Phase I: 


¢ Each college created committees to undertake MMA planning and decision-making. Committees 
included representatives from administration, English, reading, math, admissions, testing, ad- 
vising, registrar, IT, and institutional research departments. Involving different stakeholders in 
MMA redesign conversations was important, as it allowed them to weigh in on key decisions 
before implementation. Participants then had a better understanding of how the MMA initiative 
would affect their roles and responsibilities. At some colleges, existing developmental education 
committees did most of the MMA planning and decision-making. 


1. Participants’ answers were audio-recorded, and detailed notes were taken as well. Interviews were transcribed 
and uploaded to Dedoose qualitative analysis software. Researchers developed a codebook to analyze a range of 
themes such as design rationale, experiences implementing MMA, and perceived impacts of MMA on students, 
staff, and faculty experiences. 
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¢ During Phase I, colleges first selected measures that would be used to determine a student’s 
placement. Measures were evaluated based on their predictive validity (when known), availabil- 
ity, usability, and cost.? Options included traditional placement tests, noncognitive assessments, 
students’ high school grade point average (GPA), other high school transcript information, and 
other standardized test results. 


¢ Once a college decided what measures to use, it established specific combinations of the 
measures used to place students. Each college created a flow chart or map (see Figure 2.1) that 
displayed its placement rules, whether it was using decision bands to “bump up” students in a 
test score range or decision rules to define exemptions and cut-off scores for placement into de- 
velopmental and college-level coursework. In all cases, students could be placed higher based on 
the consideration of additional measures but would not be placed lower than if they were being 
placed using a single placement approach. 


¢ Each college piloted its new MMA system with incoming fall 2017 students for brief periods to 
test the feasibility of applying it more widely later. With the exception of one college, most col- 
leges ran their Phase I pilots for a one- or two-week window. During this phase, much of the work 
was manual, as staff compiled and reviewed each individual student’s relevant data and made a 
placement decision based on the college’s decision rules. 


Phase II: Continued Development of an MMA System 


To refine and scale up their MMA systems, the team at each college worked with various departments 
to consider the procedural changes required for full implementation. For instance, developmental 
and college-level faculty were included in conversations, because of the impact of MMA imple- 
mentation on their courses’ enrollments (e.g., numbers of sections needed, classroom composition, 
etc.). At one college, the communication strategy involved presentations on the project at satellite 
campuses, speaking to the curriculum committee and the president’s council and meeting with a 
group of academic advisers. The same college hosted an integrated planning day where the campus 
community could learn about MMA and ask questions. At another college, the MMA team went on 
a “listening tour” to hear about community members’ views on Phase I and share its plans for Phase 
II. Based on the feedback from this “listening tour,” the MMA team increased faculty involvement. 


One developmental education faculty member explained: 


We asked to be included, actually. They were coming around to different departments and 
we said we felt we needed to be represented because [the MMA placement system] affects 


our students. 


As the pilot MMA systems moved toward becoming automated (e.g., building rules into college 
computing systems, integrating high school GPA into placement systems, etc.), committees oversee- 
ing the initiatives were generally less involved. In fact, much of the work in Phase II was technical, 


2. Cullinan et al. (2018). 


6 | Expanding Access to College-Level Courses 


L | sula}sXg Juawssassy sainsea ajdiyjny Gurubiseg pue Guruueld 


FIGURE 2.1 Anoka-Ramsey Community College Decision Rules 


Application Waivers ACCUPLACER Exam 


Possible waivers (ACT, 


GPA and/or LASSI 
Review 


Placement 


———> SAT, college ee 


transcripts, MCAs, 
etc.) 


Scoring at or above 
current college- 
ready cut scores on 
ACCUPLACER/ 


Student applies NGA TEST 


to Anoka-Ramsey 
Community College 


Scoring below 
current college- 
ready cut scores on 
ACCUPLACER/NGA 
TEST 


GPA is 3.0 or above 
—— or LASSI score in 
MOT 2 50th percentile 


GPA is below 3.0 and 
— LASSI score in 
MOT < 50th percentile 


College-level 
placement 


College-level 
——- placement or 
up one level 


Use existing 
—_——_» ACCUPLACER/NGA 
cut score for 
placement 


involving changes to testing and IT systems as described below. During early implementation, com- 
mittees met occasionally to monitor progress and, in some cases, review data on student placement 
and course completions. 


Despite efforts at communication, not all staff felt adequately informed about Phase II. At two col- 
leges, advisers felt there was miscommunication at the beginning about how the project would unfold 
and how it would affect the work of advisers. In another case, a faculty member was not clear on the 
purpose of the new MMA approach: 


We sort of thought this was gonna be about improving accuracy all around, and then when 
it ended up being mostly about eliminating underplacement, that was where there was a lot 


of pushback from the faculty. I think if we’d just known that going in it would’ve been better. 


MMA System Designs 


The MMA system designs developed by the participating colleges included selecting measures for 
student assessment, establishing a set of decision rules for placement decisions, and developing 
systems to accurately place, advise, and register students. 


Measures Used 


In most cases, the measures used in Phase II were similar to those originally established in Phase 
I. All colleges included the following measures in their MMA systems: placement test scores, high 
school GPA, the results of noncognitive assessments, and scores from the ACT and SAT. The specific 
measures and decision rules used at each college are displayed in Table 2.1. 


Measures were chosen based on several factors, often following extensive discussions within the 
college and with MDRC, CCRC, and other participating colleges. The most important factors in 
the selection process were evidence of the predictive value of individual measures, when available, 
cost, and ease of administration of assessments. As in Phase I, colleges decided that students would 
not be placed lower when evaluated using the added measures, if their placement using a single test 
would have been higher. They would only have the opportunity to place higher or be “bumped up.” 
In addition, every college continued to use the ACCUPLACER placement test as part of their new 
placement system. 


The high school GPA was used at every college because prior research showed it is a useful predic- 
tor of student success in initial college-level math and English courses.* One college administrator 
stated that the decision to use GPA came from faculty engagement in national conversations and 
compelling research findings indicating that GPA is a strong predictor of college readiness: 


[Faculty reviewed] national literature, especially on GPA, on self-reported GPA, conference 
findings. Our faculty were actively engaged in the CADE Conference, the Conference on 


3. Belfield and Crosta (2012); Scott-Clayton (2012). 
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TABLE 2.1 MMA Approaches at Colleges in the 
Multiple Measures Assessment Study — Phase II 


COLLEGE NAME 
AND STATE 
Anoka-Ramsey 
Community 
College, Minnesota 


Century College, 
Minnesota 


Madison College, 
Wisconsin 


Minneapolis 
Community and 
Technical College, 
Minnesota 


Normandale 
Community College, 
Minnesota 


Northeast 
Wisconsin Technical 
College, Wisconsin 


TYPE OF 
PLACEMENT 
SYSTEM 


Decision rule 


Decision rule 


Decision band 


Decision band 


Decision rule 


Decision band 


MMA APPROACH 
AND ORDER OF STEPS 


1. Exemptions (AP/ 
IB, ACT, SAT, MCA 
scores) 


2. ACCUPLACER 
(exemption) 


3. GPA or LASSI 


1. Exemptions (AP/ 
IB, ACT, SAT, MCA 
scores) 


2. ACCUPLACER 
(exemption) 


3. GPA or LASSI 


1. Exemption (ACT 
score) 


2. ACCUPLACER 
(decision band) 


3. GPA or Grit Scale 


1. Exemptions (ACT, 
SAT, MCA scores, 
college credit, IB) 


2. ACCUPLACER 
(decision band) 


3. GPA or LASSI 


1. Exemptions (AP, ACT, 
SAT, MCA scores, 
college credit) 


2. LASSI 


3. GPA or 
ACCUPLACER 
(exemption) 


1. Exemption (GPA) 


2. ACT + Grit Scale or 
ACCUPLACER + Grit 
Scale (decision band) 


NONCOGNITIVE ASSESSMENT 
LASSI (motivation): 50th percentile 


LASSI (motivation): 50th percentile 


Grit Scale: 4+ 


LASSI (motivation): 75th percentile 


LASSI (motivation): 75th percentile 


Grit Scale (perseverance): 
3=1 pt; 

4=2 pts.; 

5+ = 3 pts. 


COLLEGE-READY HIGH 
SCHOOL GPA LEVEL 
English/Math: 

> 3.0 GPA 


English/Math: 
> 3.0 GPA 


English/Math: 
> 2.6 GPA 


English: 
> 2.3 GPA 


Reading: 
> 2.4 GPA 


Math: 

> 3.0 GPA 
English/Reading: 
>2.5 GPA 


Math: 
> 2.7 GPA 


English/Math: 
> 2.6 GPA 


NOTE: DECISION RULES are a sequence of rules that compares each selected measure to a threshold in a predetermined order. If the threshold 
is met, a placement is generated; if not, another rule is applied. DECISION BANDS are decision rules that apply only to students who fall within a 
certain range on a specified indicator (such as high school grade point average or a placement test score), usually just below the cutoff. 
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Acceleration in Developmental Education. And those were pretty important places where 
they kept hearing the same message over and over again that GPA was a better determiner... 


of college readiness or performance in college-level classes. 


Decisions about what high school GPA level should be used to place students into college-level courses 
were often difficult to make. One college decided on a 3.0 GPA cutoff as a compromise because some 
faculty wanted a lower GPA, while others wanted a higher GPA. Math faculty often focused on the 
adequacy and rigor of students’ high school math coursework, with specific high schools cited as 
potential sources of concern. They worried that students could be inaccurately placed by the MMA 
system if, for example, they had previously taken only one or two high school math classes and then 
were “bumped up” into college algebra based on their overall GPA. 


In addition, considerable thought went into the selection of noncognitive assessments. In Phase 
II, the Grit Scale was selected by two colleges, while the Learning and Study Strategies Inventory 
(LASSI) was used by four colleges. During Phase I, colleges had reviewed research on several non- 
cognitive assessments to understand the extent to which they predicted success in college as well as 
time students would spend in testing and the cost of the assessment options.* 


The Grit Scale measures perseverance and passion for long-term goals. It is available at no cost 
and has been shown to predict positive outcomes in college settings.° The LASSI is a much longer 
assessment that addresses factors ranging from motivation to comfort with testing. Some colleges 
appreciated the opportunity to have more extensive information about their incoming students, de- 
spite the cost to use the test and the greater amount of time students spent in testing. For placement 
purposes, colleges used only the LASSI’s motivation scale, which prior research shows is predictive 
of success in college.® 


Other standardized test scores such as the ACT and SAT served as the final widely used measure. 
Using these results for placement test waivers was already widely established and influenced by state 
legislation in Minnesota. Though waivers differed slightly by college, the tests considered for direct 
placement into college courses typically included the ACT, the SAT, the Minnesota Comprehensive 
Assessments, and International Baccalaureate exams. Some colleges also considered grades earned 
at previously attended colleges when granting waivers. 


Placement Approach 


Once measures were selected, colleges had to decide how to combine them. Most established a set of 
decision rules, a specific hierarchy of measures (starting first with a primary measure for all students 
and then adding additional measures) to determine students’ eligibility for designated classes. The 
colleges sought to automate this process as much as possible, using technology to get to full MMA 
system implementation. The fourth column in Table 2.1 shows the sequence in which each college 


4. See Cullinan et al. (2018) for more information on different noncognitive test options. 
5. Duckworth, Peterson, Matthews, and Kelly (2007). 
6. Carson (2012); Rugsaken, Robertson, and Jones (1998). 
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considered the measures it used. Typically, colleges considered test scores and credit-bearing course- 
work as waivers first to exempt students from testing before using other measures in their MMA 
systems. Next, they considered results of the ACCUPLACER, high school GPA, and noncognitive 
assessments. In some cases, they used a system of “decision bands” — rules applicable to students 
within a particular score range. Students within the band (typically placement test scores just below 
a college-ready designation) would be assessed using additional measures. 


The decision rules set by each college were sometimes very simple, with only a few rules to consider. 
(See Figure 2.1 based on rules from Anoka-Ramsey Community College.) In other cases, they were 
much more complex, particularly when determining which math courses students could take. (See 
Figure 4.1 based on rules from Northeast Wisconsin Technical College.) In a few cases, students’ 
high school math courses were reviewed as part of the placement process. 


Use of Placement Determinations 


The MMA system was mostly used to distinguish developmental education students from those 
ready to take college courses. One college contemplated using their noncognitive assessment results 
for other reasons than student placement: 


Also, and I’ve said this before, we’re using something that has a lot of information that 
we're not using as part of this pilot. The question is ... are there other ways we should be 


thinking about using this information? 


The college believes it can take advantage of the noncognitive assessment results to help identify 
appropriate supports for students. 
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Implementing Multiple Measures 
Assessment Systems 


» nce MMA system designs were finalized, colleges needed to adopt or adjust policies and 
procedures for obtaining relevant high school information and noncognitive test scores, 
~ and then integrating those data into placement determinations. Student admissions, testing, 

advising, and communications with students were often affected as well. 


Integrating Data and Testing for More Accurate 
Placement Results 


Obtaining the High School GPA 


Each college had to create procedures for collecting, recording, and incorporating high school GPAs 
into their placement systems. However, because GPA is typically collected from high school tran- 
scripts, and those transcripts are rarely required for admission to open access two-year colleges, few 
schools had a systematic process for collecting student transcripts. Importantly, since every college 
incorporated high school GPA into their MMA rules, program students with missing high school 
information lost an opportunity to benefit from MMA placement. As one administrator explained: 


[Right] now we don’t require transcripts for new students. And only about 50 percent of 
students turn in transcripts. If we need those transcripts to make these decisions around 
multiple measures, well, we better institute some ways in order to get those transcripts and 


make it easier for students to do so. 


Even colleges that routinely collected high school transcripts had to rethink their existing practices 
when creating their MMA systems. One information technology (IT) department official summed 
up some of the additional challenges in using high school GPA in MMA decisions: 


So, we do require high school transcripts for admission into our programs. We were captur- 
ing those transcripts and putting them into our enterprise content management system, 
but we weren't routinely reading them to capture the high school GPA and recording it into 
our student administration system for use. So, when we got to the point of trying to add 
that criterion for the multiple measurements placement, sometimes we had it, sometimes 


we didn’t. 
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Some colleges avoided the need for new procedures by accepting self-reported high school GPAs, 
allowing colleges to obtain these data from students without collecting or recording official high 
school transcripts. Acceptance of self-reported GPAs also sharply reduced the risk of excluding 
students unable to provide official high school transcripts. Of the six colleges included in this study, 
two accepted self-reported GPAs, and four only accepted official high school transcripts. Both col- 
leges that accepted self-reported GPAs were influenced by research indicating that students tend to 
self-report their GPAs accurately. For example, a faculty member from one college stated: 


But then again looking at other studies, the self-reported GPA is close enough to an actual 
GPA anyhow. ... [So] students don’t tend to over-report because they don’t want to get 
placed too high [particularly in math]. So, it seems to be within a reasonable margin of 
error [of] their actual GPA. 


Further, college-initiated verifications of self-reported GPAs provide evidence in support of its use 
in the placement process. Two administrators explained: 


Speaker 1: We’re basing the placement rules on self-reported GPA. So that’s based on our data 
from the pilot that students’ [self-reports were] actually pretty accurate. And, if anything, 
they underreported. So, we went ahead with this next phase of the study. 


Speaker 2: Yeah, we did correlational analysis, because we didn’t have to worry about time- 
liness. So, we got a lot of transcripts obviously later. But we were able to [compare those 
with the] self-report and that assured faculty. They were much more comfortable and they 
were okay going this way, then. 


Test Administration 


The new MMA systems required some or all of each college’s incoming students to take two tests 
for placement: the traditional ACCUPLACER exam and a noncognitive assessment chosen by the 
college. Although one college permitted students to take the noncognitive test before sitting for the 
placement exam, most students took both tests during their visit to an on-campus testing center. 
The Grit Scale items, used by two colleges, were integrated into the ACCUPLACER exam, making 
them easy to administer. However, in the colleges that used the LASSI, students needed to take two 
separate tests supported by two different testing interfaces.’ In many cases, this created additional 
demands on testing staff and students. Colleges said students often needed assistance navigating 
between testing interfaces or with reopening tests that were inadvertently closed before completion. 


Colleges enacted various strategies to minimize stress on students and staff with their testing proce- 
dures. While some colleges hired new testing staff or trained existing staff to address student needs 
within the context of MMA testing, one college shifted from a walk-in placement testing system to 


1. While it is possible to write decision rules based on students’ responses to questions within ACCUPLACER, the 
system cannot deliver or retrieve scores from outside assessments such as the Learning and Study Skills Inventory 
(LASSI), which was the noncognitive assessment chosen by four of the colleges. 
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an appointment-based model. At this college, though, the noncognitive testing procedures inevitably 
increased students’ overall time at the testing center. Testing center staff and MMA leaders raised 
concerns over the potential increase in testing anxiety and fatigue. Staff also worried that students 
might be discouraged by the results of the noncognitive assessment, and that these results could 
then negatively impact students’ views of themselves or their performance on the ACCUPLACER 
exam. According to a staff member, one father worried about the impact of the LASSI test on his son: 


He was questioning whether or not he should even be taking the [placement] test because the 
results were saying he was lacking in motivation and lacking in this and that, so he focused 
a lot on the results of the LASSI test and [the] father and the student felt that was a big dis- 
traction for him then when he was taking the math [ACCUPLACER] test because he was so 
focused on the results of what the LASSI were telling him. 


Integrating Data Points 


The use of multiple measures for placement decisions required colleges to integrate data points 
stored on disparate, independently built information technology systems. Procedures for combin- 
ing information derived from multiple systems varied by college. Some colleges decided to integrate 
data after collecting all relevant information on each student. In these instances, colleges created a 
single spreadsheet with relevant data taken from downloaded information stored in separate data 
systems. This could either be uploaded into another system responsible for calculating placement 
decisions or manually interpreted by an adviser. 


Other colleges started integrating student data before testing began. One school took advantage 
of ACCUPLACER’s preregistration module to upload student profile information before test ad- 
ministration. That allowed the college to build MMA placement rules within ACCUPLACER that 
automatically generated placements. Importantly, staff at this college needed to manually enter 
students’ noncognitive scores at the end of the testing session and rebuild the student’s score report 
for each student. For students whose placements changed after noncognitive scores were entered, 
staff had to manually change placement decisions in the institutional data system used by advisers 
and the registrar. 


When manual data integration methods were used, they took significant time to implement and 
placed additional responsibilities on staff responsible for delivering and administering the MMA 
system. Two colleges created new positions or hired new people to meet the demands of new MMA 
systems. One testing staff member said: 


There was actually a time when one person, which was me, was able to do all that, and so 
we have a lot more people involved in the placement testing process now than used to be 


in this department. 
One college developed procedures that avoided having to combine testing information across sys- 


tems or hiring more staff. There, the collection of multiple measures was embedded into the testing 
experience, eliminating the need to integrate data points separately. Students were prompted to 
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enter their self-reported GPA and their just-completed noncognitive assessment score directly into 
ACCUPLACER. Rules programmed directly into ACCUPLACER by the college immediately retrieved 
each student’s high school GPA, noncognitive test score, and ACCUPLACER score to make appropriate 
placement determinations at the conclusion of the testing session. While this process left room for 
student-induced error, it generally worked well. A staff member from the testing center explained: 


Yeah, but like I said, some other places had to hire other people. We just wanted to make 
sure it worked for anyone, and so that’s how we went our route. I mean, we could’ve hired 
someone else... But now that I have a system and have embedded so much more and increased. 


a lot of automation, it is so much faster. 


Placement Communications and Advising 


In Phase II, students didn’t receive individual placement scores, only information on the course 
or courses into which they placed. However, the amount of time between testing and placement 
communications varied. Whereas some students were placed at the conclusion of testing, others 
waited several hours or had to attend a separate advising session to receive placement decisions. This 
was due to the variations in how data were collected and integrated for multiple measures systems. 
Colleges with procedures that allowed ACCUPLACER to deliver placement decisions quickest were 
based on systems that uploaded information using the preregistration module reported directly into 
ACCUPLACER by students during testing. 


Placement communications took longer when data integration occurred outside of ACCUPLACER. 
A testing staff member described the impact on students at one college: 


Before it was, “You have to wait an hour or two for the scores to upload.” Now it’s, “By the 
end of the day, the score should be in and the next morning you can register.” We do have 


[the option of performing manual procedures] if the student needs to register right away. 


Although students at most colleges received placement decisions before meeting with an adviser, ad- 
visers at two colleges determined whether each student was eligible for a higher class level placement 
using the information collected through MMA; they then communicated the placement decision to 
the student. Importantly, these two colleges adopted different procedures for delivering these data 
points to the adviser. At one college, students were responsible for taking their test results to an 
adviser for interpretation. At the second college, the testing center created rosters with participating 
students’ information and shared it with designated math and English faculty advisers. 


While time-consuming, one of the two colleges believed using advisers for placement decisions and 
communications was better than depending on automated procedures, since it was consistent with 


their philosophy on proactive and intentional advising. As one administrator explained: 


We’ve heard the message all along that this is intended to be an automated system. That 


it’s not intended to be an advising deployed system. Our faculty and staff and advising that 
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are part of the steering committee have all felt very strongly that this should be a decision 
on the students’ part, not just a, “Oh, look what class you got.” You know? So, that’s been 


really strong. 


Factors Affecting MMA Implementation 


Colleges identified several factors that influenced the likelihood of successful implementation of 
their new MMA systems. 


Staff Involvement 


MMA team leaders emphasized that college buy-in was essential for high-quality implementation. 
Staff who understood the potential benefits of multiple measures over traditional placement methods 
were enthusiastic and motivated to ensure students were able to take advantage of the new system. 
One administrator described this relationship, stating: 


Now that we have the buy-in, and everybody agrees it’s a good thing, it’s literally, “How do 


we make sure that this can occur in the most effective and efficient way?” 


In all cases, adopting a new MMA system required new or modified policies and practices across 
a range of institutional functions, including admissions, testing, advising, and registration. When 
staff were not involved in planning activities or when lines of communication broke down, staff and 
faculty were more likely to be confused or uninformed about system procedures, which, in turn, 
could impact implementation efforts. At one college, advisers reported that their lack of involvement 
early on had longer-term consequences for their department: 


I think we were not really consulted in terms of how multiple measures was going to work. 
We were just sort of told, this is what was going to happen, and this is the pilot that we’re 
going to be doing. And then we’ve had to deal with some of the fallout for how it was 


implemented. 


On the other hand, colleges whose staff were involved in MMA design decisions and planning ac- 
tivities appeared more optimistic about their new systems. Efforts to include staff responsible for 
enacting new policies and practices in planning activities ensured they understood and followed 
new system procedures. For example, one administrator explained: 


I think we always have to talk about our climate as an institution, and that changed when 
our new president came in. [He] really believes in collaborative leadership, of walking the 
talk, making sure that all of us are at the same table having the conversations, and I think 
that’s been beneficial to this process, so that people don’t feel like we're surprising them 


with something or that we're trying to change without having them at the table with us. 
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Staff Turnover 


Changes in staffing also affected implementation of MMA systems. At one college, the MMA team 
experienced only small changes in its membership. In contrast, another college’s high rates of staff 
turnover negatively impacted MMA planning and implementation. With many staff leaving the 
college before its full implementation, it was difficult for the MMA lead to foster teamwork and 
collaboration. 


Resource Allocation 


Implementation was often more labor-intensive and time-consuming than anticipated. In some cases, 
college leadership was instrumental in mobilizing and redirecting additional institutional resources 
towards implementation efforts. One college prioritized MMA work by integrating planning, design, 
and implementation goals and timelines into the college’s strategic plan. 


Technology Capacity 


Developing and enacting the procedures necessary to implement the new MMA systems required 
each college to make its existing data systems compatible with newly adopted MMA placement sys- 
tems, such as storing and accessing multiple measures data and aligning placements with updated 
MMA decision rules. Importantly, these efforts required significant up-front effort and the ability 
to work within and across existing technologies to identify what was possible and what was not. 
This is what it was like, from the vantage point of one IT staffer: 


Some of the challenges [involved] figuring out just what we could do with ACCUPLACER. 
So how we could ... set up all the rules... And then how we get that data all together, so 
there’s only certain ways we can get information into ... the main database... So that was 
a lot of planning set up, just figuring out what the limitations to the system are and then 


double checking when we’re setting up to making sure that it would all go well on launching. 


Implementation efforts depended on staff’s capacity to work within ACCUPLACER and across exist- 
ing data systems. Differences in colleges’ knowledge of decision rule functionality within their data 
systems affected how and when placement decisions were reported to students, for example. Colleges 
using manual systems generally acknowledged that they were not ideal and said they planned to 
learn how to improve their automation. 


Regardless of any institution’s internal capacity, Minnesota colleges were confronted with 
ACCUPLACER’s inability to interact with the state’s existing information storage and retrieval 
systems (ISRS). Incompatibility between data systems complicated efforts to create more automated 
and seamless system procedures. In these instances, schools maintained manual processes to ensure 
that all data elements were available for placement decisions. As a staff member from one college 
explained: 


Yeah, we do a lot of manual workarounds right now, so that’s where Excel’s kind of the es- 
sential home base. We’ll pull data from ISRS stored in there. During testing we'll pull data 
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from the LASSI website, get it back into that main space and then try and drop everything 
into ACCUPLACER before the student finishes. 


College Collaboration and Technical Assistance 


Although specific MMA design system procedures varied across institutions, every college ben- 
efited from opportunities to interact with other schools participating in the project. Some colleges 
also received help and technical assistance from the College Board, MDRC, or outside consultants. 
Colleges said this support helped them absorb lessons learned at other institutions and develop or 
augment the skills required to integrate automated processes and strategies. Collaboration was useful 
throughout implementation, both in terms of addressing unanticipated issues and thinking about 
system improvement and sustainability. For example, one faculty member said: 


I also work with the other schools in our system that are participating in this pilot, and 
Ive been facilitating some conversations with them... [We’re thinking about piloting] the 
system for getting more high school transcripts. ... So, [we’re]trying to figure out solutions 


to or figure out which are some of the next steps to improving our process. 


Quality Assurance 


Colleges also established and maintained manual procedures to verify that systems worked as in- 
tended. Several colleges explained that these quality assurance mechanisms were necessary to identify 
incorrect placements that could result from college-created misspecifications within ACCUPLACER, 
miscommunication among the various data systems used to collect and store multiple measures 
data, data entry errors, or student-specific scenarios not previously considered. Responsibility for 
troubleshooting was frequently assigned to or taken up by an individual who was particularly knowl- 
edgeable about the college’s specific MMA system and who participated in the design and planning 
stages of MMA development. 


Fidelity of Implementation 


Faculty and staff from each college agreed that their own MMA system was implemented as intended. 
At each college there were substantial contrasts between the assessment systems used for placing 
students in the program and control groups. College staff reported that they faithfully followed 
placement rules and adhered to placement procedures for both groups of students. There were a few 
notable exceptions, though. 


Inaccurate Placements 


Colleges that automated decisions on student placements within ACCUPLACER needed to reconfig- 
ure ACCUPLACER to reflect new MMA decision rules. This work depended upon staff’s familiarity 
with Boolean logic and their ability to write accurate placement rules within ACCUPLACER. At 
one college, the MMA system did not fully operate as designed during the pilot and into the first 
few months of Phase II. More specifically, at this college, automated placement decisions within 
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ACCUPLACER were compromised due to local errors that instructed the system to pull student test 
scores from fields that did not actually hold information on students’ scores. 


Although the college figured out that ACCUPLACER was delivering inaccurate placement decisions 
during Phase I of the project, staff could not figure out why misplacements were happening until 
November 2018. As a workaround, staff in the testing center manually entered each student’s correct 
scores into the data system used by advisers when communicating placement decisions to students 
at the end of their testing session. There may have been instances where students in the program 
group received course recommendations that did not follow the college’s MMA rules, though. This 
could happen if an adviser followed the misplacements generated by ACCUPLACER or was unable 
to read or interpret information within the data system accessed during placement conversations, 
or testing staff made errors when manually recording students’ ACCUPLACER scores into the data 
system accessed during placement conversations. 


Unintentionally Excluded Students 


Limitations of the system procedures used to enact the MMA systems unintentionally excluded 
remote test-takers in at least one case.” At some colleges, students needed to take a noncognitive 
test to be eligible for placement by the MMA system. In Minnesota, colleges paid a licensing fee to 
access the noncognitive LASSI exam administered to its students. Because access to the exam was 
restricted to students using registered computers on campus, one college was unable to administer 
the noncognitive exam to students doing placement testing off-campus. Notably, remote test-takers 
enrolling in the two Wisconsin colleges would not be at risk for the same systematic exclusion be- 
cause they received a link to the noncognitive test (a Grit Scale exam) by email and completed their 
assessments prior to visiting the testing center. 


Complying with Placement 


While colleges generally implemented MMA placement procedures with fidelity, students were 
not always expected or guided to take the math and English courses they placed into during their 
first semester. Although most colleges indicated that advisers or testing staff encouraged students 
to take these classes in their first semester, enrollment behaviors could and often did vary due to 
a variety of factors, including specific program requirements, full-time student status, scheduling 
considerations, and personal preferences. Students planning to re-test for a higher placement also 
might choose to delay enrollment in these courses. Only one college required all incoming students 
to follow their math and English placements in the first term. To improve the likelihood of students’ 
overall success, another college explicitly encouraged students with low reading scores to prioritize 
reading skill development and delay math enrollment until after the first term or first year. 


2. Several groups of students were intentionally excluded from the study by design, including dual enrollment and 
English language learners (ELLs). One college also limited participation eligibility to students in science, technology, 
engineering, and math (STEM) and business groups. 
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Additionally, previously established post-placement appeal policies and processes remained in place. 
These post-placement policies allowed students to enroll in courses outside of the referred level of 
placement and, therefore, may have impacted initial placement compliance in some cases. 


Most colleges reported having an appeal process that, when successful, allowed students to enroll 
in higher level courses. One college permitted advisers to change students’ math placements by one 
level if certain faculty-designated criteria were met, while another college allowed some students to 
enroll in higher-level classes with instructor permission. Other colleges described policies allowing 
students to enroll in courses below their placement level at the student’s discretion. In all cases, college 
faculty and staff reported that historically, post-placement procedures lead to few placement changes. 


At another college, using manual overrides during post-placement registration affected placement 
compliance. For students to enroll in a higher course than their ACCUPLACER scores alone would 
allow, students needed to attend an advising and registration session to obtain a course override. 
These sessions, which were not required of all students, did not give most self-advising students the 
opportunity to enroll in the higher-level courses. 


Exemption Records 


All six colleges worked to incorporate existing waiver systems into their new MMA systems. Students 
with waivers were exempted from placement tests and immediately eligible for college-level work if 
they demonstrated college readiness through other means, such as the SAT or ACT scores, or pro- 
vided evidence that they successfully completed specific subject relevant courses in high school. Test 
score records that could exempt students from placement tests were sometimes difficult to obtain; 
this was true regardless of whether they were in the MMA group or control group. Therefore, it is 
possible that students in both program and control groups may have received placements when they 
should have been exempt from the assessment. 


The Student Experience 


As intended, students’ testing and placement experiences were, in most cases, identical regardless of 
program status. This is essential for identifying the causal impacts of MMA on student outcomes. 
At one college, however, there was a slight variation in the post-placement advising and registration 
session for students who were bumped up as a result of the MMA system. Because registration in 
the higher-level course required manual override codes provided by faculty advisers, students with 
higher placements got more personalized attention during these sessions before course registration. 


Implementing Multiple Measures Assessment Systems | 21 


Early Impact Findings 


"his chapter presents findings on the multiple measures assessment (MMA) placements’ 
estimated effects on students’ academic outcomes. The chapter describes the sample and 
summarizes the main academic effects in the semester after students were randomly as- 

signed to the program and control groups and placed into courses, and how MMA placement af- 

fected enrollment and course completion. It concludes with some considerations of future MMA 
program effects. 


Sample Intake and Characteristics 


Identifying, Recruiting, and Randomly Assigning Students 


Five of the six colleges in this study participated in the randomized controlled trial, targeting all 
students taking placement tests for enrollment in the fall 2018 semester. Colleges chose not to in- 
clude dual enrollment students taking courses at the college while in high school, as well as English 
language learner (ELL) students, because of concerns about whether high school GPAs based on 
ELL coursework would have comparable predictive value for college coursework. Across the four 
Minnesota colleges,’ 5,282 students participated in the study. There were 3,677 students testing for 
English placements and 4,487 tested in math. Students may not have had to test in both subjects if 
they had high enough ACT scores or Minnesota Comprehensive Assessment (MCA) scores or had 
eligible transfer credits. 


All 5,282 students were randomly assigned either to a program group placed using high school GPA, 
noncognitive Learning and Study Strategies Inventory (LASSI) test scores used by the four colleges, and 
the traditional ACCUPLACER placement test; or a control group, which used only the ACCUPLACER 
test.? Multiple Measures Assessments meant about 17 percent (English) and 16 percent (math) of sample 
students would be placed into a college-level course rather than a developmental course if they were 
in the program group, based on either their high school GPA or LASSI results. About two-thirds of 


1. The fifth college in the randomized controlled trial, from Wisconsin, randomized a large number of students, but 
because of implementation bottlenecks associated with a lack of automation in their placement process, a very 
small number of students were given the opportunity to be placed using multiple measures in the first program 
semester. Changes were made to improve this for the fall 2019 cohort, which will be included in the next report’s 
analysis, but for now, the fifth college is not included in the analysis. 


2. The program to control random assignment ratio was 50/50 at Normandale and Anoka-Ramsey, and 70/30 at MCTC 
and Century. Because of this, there are slight differences in the percentage of students falling into the bump up zone 
and the impact on placement into gatekeeper courses in each subject discussed below. 
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those college-level placements were because of high school GPA, the rest because of LASSI scores. 
The same percentage of control group students would have been eligible to be placed in college-level 
classes by the MMA rules in each subject had they been in the program group.® This implies that 84 
percent (English) and 82 percent (math) of students were referred to the same course regardless of 
the placement procedure used. For students whose placement is unchanged, the expectation is that 
the use of multiple measures will have no positive (or negative) effect on their academic progress. 


The breakdown of students from both research groups who place into developmental courses, college- 
level courses, or fall into zones that result in higher-level course placements is shown by subject in 
Table 4.1 below. The always developmental and always college-level rows represent those for whom 
the referral approach (MMA versus traditional) has no effect on placement. 


TABLE 4.1 Full Sample Breakdown by Subject Placement, 
Multiple Measures Assessment Study — Phase II 


SUBJECT (%) PERCENTAGE N 
English 3,677 
Always developmental placement 37.8 1,389 
Bump up zone placement 17.0 624 
Always college-level placement 45.3 1,664 
Math 4,487 
Always developmental placement 69.6 3,123 
Bump up zone placement 15.7 703 
Always college-level placement 14.7 661 


SOURCE: Placement data, test scores, high school GPA and LASSI score provided by the four Minnesota 
schools: Anoka-Ramsey Community, Century, Minneapolis Community and Technical, and Normandale 
colleges. 


NOTE: Distributions may not add to 100 percent because of rounding. 


There were 624 students who tested poorly in English and 703 who tested poorly in math, but with 
strong high school GPAs or LASSI motivation scores. This subset makes up the main analysis sample. 
These students would be referred to developmental courses under the colleges’ business-as-usual 
placement system and college-level classes under an MMA system. Within this main analysis sample, 
the intervention consists of randomly assigning students the opportunity to take college-level courses 
or requiring them to take a developmental education prerequisite first.* 


3. This was confirmed by looking at the placement tests, high school GPAs, and noncognitive assessment scores for 
both research groups at the time of enrollment. 


4. Colleges offered an appeals process for students who thought they were placed too low, but few students used it. 
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Characteristics of the Sample 


Table 4.2 presents some demographic characteristics of students in the full sample. The percentages 
shown in the table are representative of these four colleges’ entering student demographics, because 
almost all entering students are included. 


TABLE 4.2 Demographic Variables, Multiple Measures 
Assessment Study — Phase II 


MEAN PROGRAM CONTROL BOTH 

CHARACTERISTIC (%) GROUP GROUP GROUPS 
Age 

20 and under 59.4 59.8 59.6 

21-30 26.7 26.0 26.4 

31 and over 9.5 9.0 9.2 

Missing 4.4 5.2 4.8 
Gender 

Male 42.2 43.3 42.7 

Female 53.0 51.0 52.1 

Missing 4.8 5.8 5.2 
Race/Ethnicity 

Asian 9.4 7.8 8.7 

Black 19.6 21.0 20.2 

Hispanic 10.5 11.3 10.9 

White 49.1 47.7 48.5 

Other 6.0 6.0 6.0 

Missing 5.3 6.2 5.7 
Pell eligibility 

Yes 42.1 40.9 41.6 

No 34.0 34.7 34.3 

Missing 23.9 24.4 24.2 


Sample size 2,941 2,341 5,282 


SOURCE: Demographic data provided by Anoka-Ramsey Community, Century, Minneapolis Community and 
Technical, and Normandale colleges. 


NOTE: Distributions may not add to 100 percent because of rounding. 


Table 4.3 shows the sample members’ averages on the measures to be used at the time of placement. 
ACCUPLACER (Classic)® score averages are shown for each test for those who attempted each test, 


5. The previous version of the ACCUPLACER test. The current, widely used version is called Next Generation. 
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TABLE 4.3 Multiple Measures for Assessing Placements 


PROGRAM CONTROL 
OUTCOME GROUP GROUP DIFFERENCE P-VALUE 
ACCUPLACER scores* 
Arithmetic 49.0 49.3 -0.3 0.777 
Elementary algebra 60.9 60.0 1.0 0.239 
College-level math 40.4 40.2 0.2 0.876 
Reading comprehension 76.3 774 -14 0.143 
Sentence skills® 17.7 78.9 -1.2 0.253 
High school GPA (%) * 0.094 
3.5-4.0 9.9 9.9 
3.0-3.4 14.6 16.5 
2.5-2.9 20.4 18.6 
2.0-2.4 12.0 13.3 
1.9 or lower 4.9 5.5 
Missing 38.2 36.2 
LASSI score (%) 0.895 
50-100 44.4 44.8 
0-49 32.6 32.0 
Missing 23.1 23.2 
Sample size (total = 5,282) 2,941 2,341 


SOURCE: Test scores, high school GPA, and LASSI score provided by the four Minnesota colleges. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an 
intervention with zero true effect. 

To assess differences between the research groups, chi-square tests were used for categorical variables and 
two-tailed t-tests were used for continuous variables. 

4ACCUPLACER test scores can range from 0 to 120. 

Only Normandale Community College sent data for and used the sentence skills test to determine course 
placement for English. The other three Minnesota colleges used the reading comprehension test to determine 
course placement for English. 


on a scale from 20 to 120. Reading comprehension, sentence skills, and elementary algebra test scores 
of at least 75 are required for college-level placement under traditional rules. The exact cutoffs vary 
by college, but about half of new students normally place into developmental courses in English 
and reading, while 85 percent of students place into developmental courses in math at these MN 
colleges. The available high school GPAs for the sample students averaged between 2.5 and 3.0 (C to 
B), with 45 percent of students having high school GPAs of 2.5 or better. However, over 35 percent of 
the group were missing their high school GPAs. LASSI motivation scores were above 50 out of 100 
for most students who took this test, but about a quarter of the sample group did not take that test. 
There is no evidence of systematic differences between program and control groups on the tests, 
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assessments, or high school GPA at the time of placement (that is, “baseline” characteristics) in either 
the full sample or the “bump up” zone analysis samples in each subject (not shown). 


Effects of Multiple Measures Assessment 


This section presents the estimated impact of the program at the end of the first semester on the first 
cohort of students. These analyses explore whether the students offered college-course placements 
from qualifying MMA results are taking steps toward completing a college-level course. The analysis 
in this interim report does not gauge the effectiveness of the changes in the placement system on the 
course completion and credit accumulation after three semesters, the primary outcomes of interest 
in this project, but provides insights into whether the short-term outcomes indicate that students 
are on track for success in later semesters. 


Summary of Findings 


In the first semester: 


¢ As intended, colleges used MMA to place program group students in their courses, with few ex- 
ceptions. This resulted in 15 to 17 percentage points more program group students than control 
group students referred to college-level gatekeeper courses. 


¢ Program group students in the full sample were more likely to enroll in college (take one or more 
classes at the college where they tested) than control group students (2.5 percentage points more). 


¢ Program group students in the full sample also enrolled in more college-level gatekeeper courses 
than control group students (4.7 percentage points more in English; 3.9 percentage points more 
in math). 


e Students in the “bump up” zone placed into college-level English were 28 percentage points more 
likely to have completed the gatekeeper English course by the end of their first college semester 
than their control group counterparts. 


¢ Students in the “bump up” zone placed into college-level math were 12 percentage points more 
likely to have completed the gatekeeper math course by the end of their first college semester than 
their control group counterparts. 


Effects on Educational Outcomes During the First Semester 


Table 4.4 presents the academic outcomes for the entire randomized sample. The control group col- 
umn represents students placed by traditional ACCUPLACER cutoffs. In the control group, nearly 
half of students placed into “gatekeeper” English courses, but only about 10 percent did so in similar 
math courses. Not all students who were placed in the college-level gatekeeper courses took them 
in the first semester, but even fewer of those who were placed in developmental courses took them 
in the first semester. This is calculated by dividing enrollment rates by placement rates. In English, 
59 percent of those placed into a college-level course took it the first semester, while 55 percent of 
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TABLE 4.4 First-Semester College Transcript Outcomes Among All Randomized 
Students, Multiple Measures Assessment Study — Phase II 


PROGRAM CONTROL 
OUTCOME (%) GROUP GROUP DIFFERENCE P-VALUE 


English 
Placed into gatekeeper course 61.8 47.3 145 *** 0.000 
Enrolled in gatekeeper course 32.8 28.1 47" 0.000 
Completed gatekeeper course (C or higher) 20.7 19.2 1.6 0.158 
Failed gatekeeper course 6.1 4.6 1.5 ** 0.015 
Withdrew from gatekeeper course 3.7 2.8 0.9 * 0.066 
Placed into developmental course 38.2 52.7 -14.5 *** 0.000 
Enrolled in developmental course 11.6 14.9 -3.3 *** 0.000 
Completed developmental course (C or higher) 7.3 10.3 -3.1 *** 0.000 
Failed developmental course 2.3 2.6 -0.3 0.430 
Withdrew from developmental course 1.3 0.8 0.5 * 0.073 
Sample size among English test-takers? (total = 3,677) 2,091 1,586 

Math 
Placed into gatekeeper course 26.3 9.4 16.9 *** 0.000 
Enrolled in gatekeeper course 9.1 5.2 3.9 *** 0.000 
Completed gatekeeper course (C or higher) 5.2 3.3 1.9 *** 0.001 
Failed gatekeeper course 1.3 0.6 0.7 ** 0.012 
Withdrew from gatekeeper course 1.5 0.8 0.7 ** 0.014 
Placed into developmental course 68.8 86.7 17.9 *** 0.000 
Enrolled in developmental course 19.6 21.0 -1.4 0.219 
Completed developmental course (C or higher) 10.7 13.6 -3.0 *** 0.001 
Failed developmental course 5.8 5.2 0.6 0.326 
Withdrew from developmental course 2.3 1.7 0.5 0.173 

Sample size among math test-takers? (total = 4,487) 2,462 2,025 

Enrolled in any course 81.1 78.6 2.5 ™ 0.021 

Total sample size (total = 5,282) 2,941 2,341 


SOURCE: Transcript data provided by Anoka-Ramsey Community, Normandale, Century and Minneapolis Community and 
Technical colleges. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with 
zero true effect. 

“As some test-takers placed in both English and math, the sum of the subject sample size numbers is greater than the total 
sample size. 
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students placed into a math course did so. However, only about one-fourth of those placed into de- 
velopmental courses in each subject took them in the first semester. This means almost 60 percent 
of the control group (status quo) English testers and 75 percent of math testers are not enrolled in 
either the college-level or developmental course in those subjects.® 


As noted above, the new placement rules did not change course placements for most program group 
students, as expected. Colleges expected to bump up between 10 and 20 percent of students in each 
subject from developmental to college-level based on the use of multiple measures. In the program 
group, 15 percentage points more students placed into gatekeeper English and 17 percentage points 
more placed into gatekeeper math than in the control group— as shown by the “Placed into gate- 
keeper” rows under each subject. 


In the full randomized sample, placement using MMA caused 4.7 percentage points more students to 
enroll in a gatekeeper English course and 3.9 percentage points more students to enroll in a gatekeeper 
math course than the control group. Slightly more students completed gatekeeper courses in the 
program group (although not significantly so for English), and slightly more students completed 
developmental courses in the control group. There was a small positive effect on overall enrollment 
as well, as shown in the last row. 


However, because so many students were ineligible for college-level course placement, they were 
not affected by the MMA rules. The study’s main intent is examining the sample of those who were 
placed in college-level courses because of the MMA rules. The program students who were placed in 
college-level courses and their counterparts in the control group who would also have been placed 
in college-level courses make up the primary analysis sample and are the focus of Tables 4.5 and 4.6, 
which show their first-semester academic outcomes. 


All students in both research groups in these tables had ACCUPLACER scores below the necessary 
cutoffs for the college-level course, which would have placed all of the students into developmental 
courses under the control group rules. However, all students in both research groups in these tables 
also had high school GPA or LASSI motivation scale scores that exceeded the MMA cutoffs at their 
colleges. This means that in this analysis sample all program students were placed into college-level 
courses and all control students were placed into developmental courses in Tables 4.5 and 4.6. 


Students in the bump up zone for college-level English placements are shown in Table 4.5. It shows 
that 10 percent of control students took gatekeeper English without qualifying ACCUPLACER scores, 
and only about a third of control group students took the developmental course, even though they 
were placed in developmental English. This has important implications for the longer-term impacts 
of this study. With such a small proportion of control group students assigned to developmental 
English actually starting developmental English in their first semester, an upper bound is set on the 
number that could enroll in and pass the college-level course in the subsequent semester: a limit of 
29 percent — if all those who passed developmental English register for the college-level course the 


6. Asmall percentage of these students may have enrolled in higher-level college courses. 
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TABLE 4.5 First-Semester College Transcript Outcomes 
Among Students in the English "Bump Up" Zone, 
Multiple Measures Assessment Study — Phase II 


PROGRAM CONTROL 
OUTCOME (%) GROUP GROUP DIFFERENCE P-VALUE 
English 
Placed in gatekeeper course 100.0 0.0 100.0 *** 0.000 
Enrolled in gatekeeper course 54.8 9.8 45.0 *** 0.000 
Completed gatekeeper course (C or higher) 34.5 6.7 27.8 *** 0.000 
Failed gatekeeper course 12.7 11 i Py aa 0.000 
Withdrew from gatekeeper course 3.2 1.7 1.6 0.223 
Placed in developmental course 0.0 100.0 -100.0 *** 0.000 
Enrolled in developmental course 5.3 36.8 -31.5 ** 0.000 
Completed developmental course (C or higher) 4.3 29.4 -25.1 *** 0.000 
Failed developmental course 0.3 2.8 -2.5 *** 0.006 
Withdrew from developmental course 0.5 2.2 -1.7 * 0.051 
Enrolled in any course 83.0 75.2 77 ™* 0.018 
Sample size (total = 624) 363 261 


SOURCE: Transcript data provided by Anoka-Ramsey Community, Century, Madison Area Technical, Minneapolis Community 


and Technical, and Normandale colleges. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 


Distributions may not add to 100 percent because categories are not mutually exclusive. 

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with 
zero true effect. 


next semester and pass it. Even in that very unlikely event, fewer students in the control group will 
have passed the gatekeeper course next semester than in the program group. 


In the program group, only 55 percent enrolled in college-level English, despite placing into it through 
the multiple measures assessments. Very few program students took the developmental course (be- 
cause they were not placed in it). Since more program students took the college-level course, more 
of them completed it (28 percentage points more than the control group) and more of them failed 
it, which was to be expected. 


There was also a significant impact on overall college enrollment for those bumped up into gatekeeper 
English, shown in the last row of Table 4.5. This indicates the placement into the college-level course 
not only affects enrollment in that subject, but that these students are about 8 percentage points more 
likely to enroll in college that semester if placed into college-level English. This suggests possible 
unintended consequences from developmental English course placements on enrollment behaviors. 
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TABLE 4.6 First-Semester College Transcript Outcomes 
Among Students in the Math "Bump Up" Zone, 
Multiple Measures Assessment Study — Phase II 


PROGRAM CONTROL 
OUTCOME (%) GROUP GROUP DIFFERENCE P-VALUE 
Math 
Placed in gatekeeper course 100.0 0.0 100.0 *** 0.000 
Enrolled in gatekeeper course 26.6 2.9 23.7 *** 0.000 
Completed gatekeeper course (C or higher) 13.1 1.6 11.5 *** 0.000 
Failed gatekeeper course 3.7 0.4 3.3 *** 0.003 
Withdrew from gatekeeper course 6.6 0.8 5.8 *** 0.000 
Placed in developmental course 0.0 100.0 -100.0 *** 0.000 
Enrolled in developmental course 44 27.4 -23.3 *** 0.000 
Completed developmental course (C or higher) 2.5 20.5 17.9 ** 0.000 
Failed developmental course 1.2 4.8 -3.6 *** 0.004 
Withdrew from developmental course 0.2 1.4 =12.* 0.075 
Enrolled in any course 86.1 82.8 3.3 0.228 
Sample size (total = 703) 358 345 


SOURCE: Transcript data provided by Anoka-Ramsey Community, Century, Madison Area Technical, Minneapolis Community 
and Technical, and Normandale colleges. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

Distributions may not add to 100 percent because categories are not mutually exclusive. 

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with 
zero true effect. 


Similar patterns are observed across outcomes in Table 4.6 for the bump up zone that elevates placement 
to college-level math, albeit with lower base rates for enrollment in that college-level course. Only 
3 percent of the control group took that course (they did not place into it). More program group 
students passed the college-level math course, by 12 percentage points, compared with the control 
group. Again, only 27 percent of the control group took their assigned math developmental course, 
with just 21 percent passing it. 


In the program group, 27 percent took the college-level math course into which they placed. While 
being bumped up in English increased college enrollment, there was not a significant difference in 
overall college enrollment rates caused by being bumped up to college-level math. This suggests 
that students may be less discouraged by developmental placement in math than in English. Perhaps 
being an underprepared math student is perceived less negatively by these students than is being an 
underprepared English student. It may also be that gatekeeper English is required as a prerequisite 
for more courses than is gatekeeper math. Notably, over 65 percent of students in both groups put 
off taking math altogether. 
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Dividing the percentage of students passing a course by the percentage of students enrolling in the 
same course yields its pass rate.’ Among those in the program group who were bumped up, 55 percent 
took the college-level English course and about 35 percent passed it. This yields a 64 percent pass rate 
in English (35 percent of 55 percent). The same calculation yields the math pass rate of 49 percent 
(13 percent of 27 percent).® These pass rates may be relevant to instructors, some of whom expressed 
concern that MMA allowed students with lower placement test scores into their classrooms. 


A representation of what might be perceived by instructors as the “status quo” pass rate can be 
calculated from Table 4.4, which includes the entire control group sample of students placed directly 
into college-level courses. Compared to the status quo, 68 percent pass rate in English and 63 percent 
pass rate in math, the bump up pass rate is 4 percentage points lower for English, and 14 percentage 
points lower for math. However, the effect on the overall course pass rate would be much smaller.? 
While the English pass rate for MMA test group students taking the college-level course isn’t far 
from the status quo rate, for math it is enough lower to suggest that those students who are bumped 
up are not performing quite as well in college math as those placed in those courses by traditional 
measures, at least among students taking it in their first semester. It remains to be seen how students 
who are bumped up compare after their counterparts placed in developmental courses in their first 
semester have had an opportunity to eventually attempt college-level courses. At that point the study 
examines the combined effect of enrollment rates and pass rates on overall success rates. 


Case Study: MMA at Northeast Wisconsin Technical College 


Undertaking Multiple Measures Assessment 


Northeast Wisconsin Technical College (NWTC), located in Green Bay, Wisconsin, joined the Great 
Lakes Multiple Measures Project in 2017, one of 10 Wisconsin and Minnesota colleges that participated 
in Phase I. As a member of this cohort, they worked closely with researchers from MDRC and CCRC 
to develop and pilot a system that would allow them to consider a number of measures to assess 
student readiness for college-level courses. NWTC offers more than 100 associate degree, technical 


7. If we assume that MMA affects outcomes only through its effect on enrollment in college-level courses, and that 
there are no students who would always defy their placement whether it was made through MMA or traditional 
methods, then the ratio of the difference in course completion to difference in course enrollment is the Complier 
Average Causal Effect of the intervention. For this completion outcome, it is 62 percent for English and 49 percent 
for math among those induced to take the course by the program. These are impacts of the program among those 
who received the treatment, whereas the impacts in the tables are among those who were offered the treatment. 


8. Pass rates for the control group in the same rows of these tables represent the percentage of those in the bump 
up zone who took the gatekeeper course, despite being placed in a developmental class, and who passed it (66 
percent in English and 55 percent in math). This represents a small, very unusual subset of students who somehow 
managed to defy their developmental placement and enroll in gatekeeper courses anyway. This does not represent 
what faculty normally perceive as the status quo. The comparison of program group bump up pass rates to control 
group full-sample pass rates is non-experimental, but descriptively addresses one of the most common questions 
about MMA: will gatekeeper course pass rates go down? 


9. The bump up enrollments comprise 18 percent of students placed into the gatekeeper English course who enroll in 
that course the first semester, meaning that the overall pass rate would be little changed at 67 percent. For math, 
bump up enrollments comprise 47 percent of students placed into gatekeeper math who enroll in that course the 
first semester, making the overall pass rate 56 percent. 
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diploma, and apprenticeship programs plus over 80 certificate programs. In fall 2017, 11,488 students 
were enrolled, of whom 77% attended part-time. The three-year graduation rate for full-time students 
is 44% (2017), considerably higher than that of many of their peers.'° 


They opted to scale up the new system to all incoming students the year after it was developed, rather 
than gradually implementing, a reflection of their internal culture. One administrator shared: “[We] 
talk about student success a lot. The culture here is a positive for innovation and trying new things. 
It’s also part of the reason why we go to scale.” 


Setting the Stage 


According to staff interviewed, NWTC is able to undertake rapid and multifaceted change because 
it has developed a culture in which it is the norm to do so, and because it has systems that facilitate 
planning and implementation of new initiatives." As it set out to re-make its assessment system, a 
large committee was formed and was given a charge or “charter” (Box. 4.1). A college administrator 
explained: 


So, we develop a charter.... Then, each of our charters has a sponsor. In this case, the 
sponsor was the president of the college. So, essentially what we do is, we talk about what 


we're going to do, what the boundaries are [and] try to prevent a little bit of mission creep. 


BOX 4.1 
MMA Charter Aim 


The purpose of this cross-functional work team is to guide the development and testing 

of a triage approach, which may include the use of GPA, prior learning experience, and/or 
noncognitive factors of assessment that will provide a more equitable placement system for 
NWTC students. 


The committee was jointly led by the vice presidents of student affairs and academic affairs. They 
reviewed college data and a variety of research reports and gathered information from peer institutions 
that had developed MMA systems. This group also led an effort to communicate within the college 
via a “road show” with various departments to ensure widespread understanding of the rationale for 
changing placement practices and to get input into the measures and procedures that would be used. 


10. Integrated Postsecondary Education Data System (n.d.). 


11. In addition to piloting MMA, NWTC was in the process of implementing guided pathways, co-requirement courses, 
the Next Generation ACCUPLACER, learning communities in English, and multiple math pathways during the same 
period. 
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College representatives widely praised the inclusiveness of the decision-making process and com- 
mented on the extent to which the project fostered collaboration. One adviser shared very positive 
feedback: 


I’ve never collaborated more with [academic departments] than we have with multiple 
measures, because really, they have included us in the conversations of what those place- 


ment scores should be, seeking our input ... on what impacts it will have on the student. 


The MMA System 


The college developed an MMA system that used a set of decision rules to make placement determi- 
nations (Figure 4.1). The measures considered included students’ high school GPA, placement exam 
scores, and the results of a noncognitive assessment, the Grit Scale. From the outset, it was intended 
that the additional measures could only be used to raise a student’s final placement, not to lower 
it. Additionally, high school coursework was considered for students entering certain programs. 
While key decisions were made early on about the MMA approach, and the system was scaled to 
all students, the college refined the process throughout the first year of implementation, leading to 
some anxiety among frontline staff who had to interpret these changes for students. 


The core decision rules, shown in Box 4.2, are similar to other colleges’ rules. However, several key 
decisions made about how to place students were not typical of colleges that have traveled this road 
in the past: 


e At NWTC, the high school GPA is taken into consideration without regard for the length of time 
students have been out of high school. 


¢ For the most part, students are not prohibited from taking college courses in other subjects even 
if they are not considered college ready in math or English. 


¢ Students with at least 15 college credits and a college GPA of 2.6 are considered college ready 
regardless of their initial placement determination. 


Like many other colleges, NWTC wanted to create a system that would be highly automated and 
would not noticeably change the student experience. As one administrator said, “I think the most 
challenging piece for us was that we really wanted the experience to be seamless for students.” 


With the help of a consultant, they embedded the 8-item Grit Scale into the ACCUPLACER test and 
created systems to generate a new, understandable placement report for students. They also made 
sure that placement data could be exported directly into the college’s student information system 
and developed “a massive long cheat sheet” for advisers to advise students with varied backgrounds 
and needs. A considerable amount of time was devoted to working through, and troubleshooting, 
possible scenarios, which one administrator said made a huge difference in refining the system: 


We did lots of practice test cases of students and things and that was helpful. ...[We] just 


really needed to get into the weeds with it and find out, “What about this student, what 
about these students, or what about this?” 
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FIGURE 4.1 Northeast 
Wisconsin Technical College 
Decision Rules 
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BOX 4.2 
Summary of MMA Decision Rules 


1. New students with specified cut scores on the ACT, SAT, and Test of Adult Basic Education 
are placed into college courses. 


2. Students with a high school GPA of 2.6 (or 3.0 for some health-related programs) are placed 
into college courses. 


3. Students with a GPA below 2.6 take the ACCUPLACER and Grit Scale assessment. 


4. Students are placed in one of three bands: college-level courses, college-level courses with 
a co-requisite course, or developmental courses. 


5. For entry into some programs, high school transcripts are read to direct placement into 
program-relevant math courses. 


Changes Attributable to MMA 


According to interviewees, the work required to establish the new MMA approach was considerable 
and required multiple iterations to achieve a fully functioning system. However, once in place, the 
new processes required little, if any, additional staff time. Further, both students and the college saw 
benefits. Fewer students now take developmental education courses and the college is spending less 
on ACCUPLACER tests, thanks to large numbers of students with high school GPAs that exempt 
them from testing. 


College representatives indicated that there have been some changes in classroom composition, 
which were generally seen to reflect improved placement accuracy. It also resulted in greater aca- 
demic homogeneity within classes, with more struggling students concentrated in developmental 
courses. An adviser explained: 


At the student level, I would say that the advantages that I’ve seen in the developmental ed 
population is, I think students are placed where they should be now. The disadvantage [is 
that some of the lower-tier classes [have many students who] need extra support. It’s taken 
the model students out of those courses and it’s made it really tough and really magnified 


the developmental issues that are going on in that classroom. 


And a faculty member remarked: 
The students who are in the [developmental education] courses now need the course. I can’t 


say since we’ve done multiple measures that I’ve had a student that I thought was misplaced. 


Which, I think is pretty, pretty huge. 
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Moving Forward 


In sum, NWTC is pleased with their decision to fully and rapidly scale the MMA system, and to 
undertake this initiative while also implementing numerous other reforms at the college. They believe 
that they can manage this because of their emphasis on “keeping students at the center,” as well as 
their systems for change management and their use of data to monitor progress. Over time, they 
will revisit their decisions, such as the use of the 2.6 high school GPA cut score and the inclusion 
of the Grit Scale. However, the school’s representatives clearly feel confident that the direction they 
are going is the right one for the college and for students. 
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S 


Implications for Practice 
and Future Research 


~\ reating and adopting alternative placement systems required several steps, including plan- 

: ning and designing the MMA system, and then developing procedures needed for successful 

© implementation. Interviews with faculty and staff at all six colleges showed implementation 

to be a complex undertaking at each school. This work required significant time and was completed 

by committees made up of representatives from across the colleges, including administration, faculty, 
admissions, testing, advising, registrar, information technology, and other departments. 


In all cases, colleges used the following measures in their MMA system: ACCUPLACER test scores, 
high school GPA, the results of a noncognitive assessment, and scores from the ACT and SAT. However, 
specific choices about how to combine the different measures to place students varied. Each college 
piloted their new MMA system with incoming fall 2017 students for a limited time period. During 
this phase, much of the multiple measures work was manual. 


In Phase II colleges made efforts to extend MMA placement to a larger group of students. Although 
some colleges made minor changes to their MMA designs, most of the work carried out in Phase 
II involved adopting or adapting policies and procedures that permitted efficient collection and 
integration of multiple data into placement decisions and communications. Using technology to 
automate these procedures allowed colleges to offer MMA assessments to a much larger group of 
students than if they relied on mostly manual procedures. In many cases, colleges needed technical 
assistance to implement system procedures. 


For colleges considering employing MMA with large numbers of students, some lessons are evident 
from the experiences of the six colleges in this study: 


¢ Communications about MMA systems before and during implementation can be critical to suc- 
cess. Messaging centered around a goal of improving the overall accuracy of placements may 
prove effective at garnering support from stakeholders, including faculty and students. 


e There is a tradeoff between more automated placement systems and more personalized pro- 
cesses found in systems that depend on interaction with advisers. Colleges in the study were 
moving toward greater automation, something that was encouraged in this project; however, they 
also were thinking about how to preserve opportunities for meaningful interactions between 
students and advisers. 
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¢ While the amount of staff time required to set up an MMA system is substantial, it diminishes 
with time and may even result in staff time savings once greater automation is used in placement 
decisions. 


¢ Timely access to high school GPA information remains a primary challenge in creating accurate 
MMA systems. In most cases, MMA implementation depends on students bringing transcripts 
to the college at the time of admission; however, this may not be the norm at some colleges. Some 
colleges are obtaining transcript data directly from local high schools, facilitating access to student 
data. It may also make sense to use student self-reports, given increasing evidence that students 
report their GPAs accurately. 


¢ There are difficulties involved in administering more than one test during the placement process. 
It is important to weigh the added difficulty of using a noncognitive assessment against its added 
value to the placement process. More information about the contribution of noncognitive assess- 
ments to better student placement determinations will be available in the final report. 


Early impacts show that so far, the use of multiple measures assessments is accomplishing what was 
planned for the first semester: changing students’ placements when they have high school GPA or 
LASSI scores above the cutoff, allowing their enrollment into college-level courses. This is particularly 
evident in the bump up analysis sample, in which students having met the school’s multiple measures 
criteria of a specific high school GPA cutoff or LASSI motivation, but not the ACCUPLACER test 
score threshold, were effectively randomly assigned to being placed into college-level courses. 


In the first semester: 


¢ As intended, colleges used MMA to place program group students in their courses, with few ex- 
ceptions. This resulted in 15 to 17 percentage points more program group students than control 
group students referred to college-level gatekeeper courses. 


¢ Program group students in the full sample were more likely to enroll in college (take one or more 
classes at the college where they tested) than control group students (2.5 percentage points more). 


¢ Program group students in the full sample also enrolled in more college-level gatekeeper courses 
than control group students (4.7 percentage points more in English; 3.9 percentage points more 
in math). 


¢ Students in the “bump up” zone placed into college-level English were 28 percentage points more 
likely to have completed the gatekeeper English course by the end of their first college semester 
than their control group counterparts. 


¢ Students in the “bump up” zone placed into college-level math were 12 percentage points more 


likely to have completed the gatekeeper math course by the end of their first college semester than 
their control group counterparts. 
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Effects on Educational Outcomes After the First Semester 


The next and final report will present an analysis of transcript outcomes from three semesters of 
follow-up and will add two more cohorts to the research sample. That follow-up will enable com- 
parisons between groups after students who placed into developmental courses have had a chance 
to complete them and enroll in college-level courses. At that point, more findings will be available 
about which placement system helps students perform better.’ 


1. The findings presented in this report are preliminary (and not “confirmatory”). The prespecified confirmatory 
outcomes on which the effectiveness of the program will be judged will be measured after three semesters, 
including two additional cohorts, and will be presented in the final report in 2021. These outcomes will include 
completion of first college-level course (C or higher) within three semesters, by subject, and cumulative college-level 
credit accumulation within three semesters. 
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MDRC seeks to learn not just whether a program is effective but 
also how and why the program’s effects occur. In addition, it tries 
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research — in order to build knowledge about what works across 
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and best practices are shared with a broad audience in the policy 
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the media. 
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ever-growing range of policy areas and target populations. 
Once known primarily for evaluations of state welfare-to-work 
programs, today MDRC is also studying public school reforms, 
employment programs for ex-prisoners, and programs to help 
low-income students succeed in college. MDRC’s projects are 
organized into five areas: 


e Promoting Family Well-Being and Children’s Development 
e Improving Public Education 

e Raising Academic Achievement and Persistence in College 
e Supporting Low-Wage Workers and Communities 

¢ Overcoming Barriers to Employment 


Working in almost every state, all of the nation’s largest cities, 
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ects in partnership with national, state, and local governments, 
public school systems, community organizations, and numerous 
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